WP: Visual Predictive Checks in Bayesian Workflows

Teemu Säilynoja

2023-07-18

Visual Predictive Checks

Gelman et al. (2020)

A: Density plots
Gabry et al. (2019) focus mostly on checks for continuous data.

B: Summary statistic
Another common visual PPC. Already more customised built.

C: Discrete observations
Very common, but fewer tools for effective visual checks in use.

D: Data split into groups
Uses tools from A - C.

In Bayesian Workflows

  • Predictive checks present on many stages of Bayesian Workflows

  • Early stages of model building can be very exploratory

  • Clear guidelines reduce ad-hoc decisions during the exploration and assessment

    \(\Rightarrow\) less mistakes

Our work

We provide structured recommendations on which visual checks to use.

  1. Continuous data
    • Truly continuous
    • Bounded
    • Point masses
  2. Counts
    • Large domain
    • Small domain
  3. Discrete with small domain
    • Bernoulli trial
    • Categorical
    • Ordinal

:::::

Continuous data

Density plots and quantile dot plots

Density plots show a KDE fit to the data.

  • When observation is not smooth, KDE can have large local error.
  • Automated goodness-of-fit test to detect issues.
    • We use our earlier graphical test (Säilynoja, Bürkner, and Vehtari (2022)).
  • When issues detected use quantile dot plots (or histograms).
    • Introduced by Kay et al. (2016).

Counts

Density plots and rootograms

  • When the range of observed values is large, continuous density plots work well.
  • As range narrows, density plots mislead.
    • In this case, rootograms are more practical.
    • We modify these to emphasize the discreteness.
  • Again, goodness-of-fit testing can tell, when density plot fails.

Small discrete domains

Reliability diagrams

Currently, bar plots are most commonly used.

  • Limited information even for a general visualization.
  • Reliability diagrams give insight into possible calibration issues.
    • Obervations are transformed to conditional event probabilities by assuming monotonicity with model predictions and using a pool-adjacent-violators algorithm.
    • Algorithm proposed by Dimitriadis, Gneiting, and Jordan (2021).

Conclusion

Visual predictive checks

  • Important part of Bayesian Workflows.
  • Structured recommendations reduce mistakes.
  • Density plots
    • Don’t forget to assess goodness-of-fit to data.
  • Discrete data
    • Reliability diagrams offer a good default.

Current stage

  • Finalizing a submission to Journal of Visualization and Interaction (JoVI)
    • open access and open review.
    • Experimental track offers a chance to include interactivity.

Visual Predictive Checks

Prior predictive checks

Effect of hyperparameters in predictions of a GP by Gelman et al. (2013)

Prior elicitation. Exposure to air pollution by Gabry et al. (2019)

  • Check for conflicts between prior predictive distribution and domain knowledge.

References

Dimitriadis, Timo, Tilmann Gneiting, and Alexander I. Jordan. 2021. “Stable Reliability Diagrams for Probabilistic Classifiers.” Proceedings of the National Academy of Sciences 118 (8): e2016191118. https://doi.org/10.1073/pnas.2016191118.
Gabry, Jonah, Daniel Simpson, Aki Vehtari, Michael Betancourt, and Andrew Gelman. 2019. “Visualization in Bayesian Workflow.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 182 (2): 389–402. https://doi.org/10.1111/rssa.12378.
Gelman, Andrew, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. 2013. Bayesian Data Analysis. Chapman; Hall/CRC.
Gelman, Andrew, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. 2020. “Bayesian Workflow.” arXiv:2011.01808 [Stat], November. http://arxiv.org/abs/2011.01808.
Kay, Matthew, Tara Kola, Jessica R. Hullman, and Sean A. Munson. 2016. “When (Ish) Is My Bus?: User-Centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems.” In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5092–5103. San Jose California USA: ACM. https://doi.org/10.1145/2858036.2858558.
Säilynoja, Teemu, Paul-Christian Bürkner, and Aki Vehtari. 2022. “Graphical Test for Discrete Uniformity and Its Applications in Goodness-of-Fit Evaluation and Multiple Sample Comparison.” Statistics and Computing 32 (2): 32. https://doi.org/10.1007/s11222-022-10090-6.